There are three ways to knockout an object successfully.
1. Turn the object color to black in the knockout areas, as well as make it
transparent. (This is the method I suggested before).
2. Have the original object color be black, and only texture map the color in
the non-knockout areas (by using a color map with stencil).
3. Texture map the ambient channel to be the opposite of the transparency.
Easy to do if the alpha channel is white where it should be solid. Then the
alpha can be applied to the Ambient & Transparency, with the Invert
Transparency applied. (This makes it completely independant of the color in
the map).
Mapping Transparency part 1
Creating a cutout with a transparency map is a little harder than it sounds, and the best way to do it is actually a little backwards from what you'd think at first. Making the edges of a polygon transparent is pretty easy, unfortunately all the other attributes of the object are still on: Ambient, Diffuse, etc. So you get a shaded transparent area which looks a little ghosty. There are two solutions:
One, make the original object color black, and texture map the color of the area you want to be visible. The only trick with this method is that you must texture map the full transparent color (on the right column) instead of the transparency value (on the left column), because the value would mix with the object color (black) and wouldnt be transparent at all.
The second option is to make a opacity mask (white where its solid), and apply this map/mask to all the values (ambient, diffuse, reflectivity, and transparency), and then select reverse transparency on the map screen. This will turn off diffuse and ambient in the fringe where you want the object to be invisible.
Mapping Transparency part 2
Applying a texture to the transparency on the left column mixes with the transparency instead of the object color, and secondly, if an object is transparent without refraction and is up against the background, the alpha will be set properly.
Mapping Transparency part 3
So to get an object to disapear, the object color must be black in
the area that is to be transparent. Usually done with an RGB texture
applied to the object color, or just a black picture with an alpha
channel ON in the dark areas. (with ALPHA also used for transparency,
and transparency color independent of object color).
